Point-and-Click Control of Unmanned, Autonomous Vehicle Using Omni-Directional Visors

ABSTRACT

The proposed method outlines a new control mechanism well-suited for small, unmanned aerial vehicles traversing in a GPS-denied areas. It has the strong advantage of simplifying the interface, so that even an untrained operator can handle the difficult, dynamic problems encountered in closed quarters. The proposed system seamlessly integrates point-and-click control with way-point navigation, in an intuitive interface. An additional advantage of the proposed system is that it adds minimal hardware to the payload of the UAV, and can possibly, strongly diminish the bandwidth and delay effects of the communication channel.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Patent Application Ser. No.62/129,471, entitled “Point-and-Click Control of Unmanned, AutonomousVehicle Using Omni-Directional Visors”, filed on 6 Mar. 2015. Thebenefit under 35 USC §119(e) of the U.S. provisional application ishereby claimed, and the aforementioned application is herebyincorporated herein by reference.

SEQUENCE LISTING OR PROGRAM

Not Applicable

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to control of autonomousvehicles. More specifically, the present invention relates to control ofautonomous vehicles using omni-directional visors.

BACKGROUND OF THE INVENTION

Outdoor control of UAVs is normally accomplished using GPS. Usually, theoperator has a map of the area where he/she is interested in sending theUAV. By selecting a series of waypoints on the map, it delineates thetrajectory followed by the UAV. This trajectory is usually in twodimensions, and it assumes that GPS is available throughout theexecution of the plan. The operator then decides whether the UAV shouldland or loiter at the end of the trajectory.

If GPS is jammed or not available, the current state-of-the-art—forsmall UAVs—is to teleoperate. Larger UAVs (like the Predator) arecapable of maintaining localization for longer periods of time, due tothe accurate, expensive, and heavy inertial navigation units they carry.On the small UAVs, this is not a choice. The MEMS-based inertial units(which fit the SWAP of the small vehicles) have enough inertial biasesthat they are not capable of flying without GPS, or at least not withsufficient accuracy. Therefore, teleoperation is the customary fall-backcontrol methodology.

Teleoperation can be done two ways; one way is when the operator hasdirect line of sight (usually called remote control). This method isperformed when the operator looks directly at the flying vehicle, anduses a joystick to control its position—as well as counteract theeffects of wind and aerodynamics. A second mechanism, usually called FPV(First-Person View), is used when the operator controls through anonboard camera, which is then relayed through a communication channel tothe OCU (Operator Control Unit) carried by the operator.

For indoor applications, the choices are more limited. GPS is notavailable, and the UAVs capable of navigating in indoor scenarios cannotcarry these larger, accurate IMUs. Therefore, the most common techniqueused for indoor missions is vehicle teleoperation. Teleoperation indoorsis not trivial; the proximity of walls, and even the ground itself,create aerodynamic effects, which—in some cases—severely affect thecontrols of the UAV. Therefore, only trained operators can be used, andeven under those conditions, safe control of the UAVs is not alwaysaccomplished.

Although autonomous mobility is the “Holy Grail” of autonomous roboticcontrol in indoor and underground facilities, this is still to beaccomplished. There are two main issues keeping autonomous mobility frombeing widespread. One, the sensors necessary for providing full,autonomous mobility, in an indoor facility, and are expensive and heavy.Two—and most importantly—the localization techniques for indoornavigation are hampered by the reduced SWAP. If a sufficient number ofsensors is added to a quadrotor, capable of accurately mapping andlocalizing in an indoor facility, the cost and size of the UAV tends tomake it unviable from a tactical standpoint.

SUMMARY OF THE INVENTION

The proposed system is a point-and-click control method, where theoperator selects a point in the image for the quadrotor to traverse. Theoperator selects a distance, and the quadrotor will traverse a straight(or curved) line between its current location and the point selected inthe image. In the proposed system, the point selection is performedusing a set of virtual reality stereo goggles (i.e. Oculus Rift). Theoperator is given an omni-directional image, where—by moving his/herhead around—he/she can observe the area surrounding the UAV. The gloveis used as a virtual mouse, selecting the point where the operatordesires the UAV to go, as well as a selecting how far, in thatdirection, the quadrotor should automatically move.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and form a partof the specification, illustrate the present invention and, togetherwith the description, further serve to explain the principles of theinvention and to enable a person skilled in the pertinent art to makeand use the invention.

FIG. 1a illustrates the interface, through virtual reality stereogoggles, that display omni-directional field of view, as well as anintegrated, blue force tracker-like map where the operator is lookingforward;

FIGS. 1b-1c illustrate the interface, through virtual reality stereogoggles, that display omni-directional field of view, as well as anintegrated, blue force tracker-like map where the operator is lookingdown; and

FIG. 2 illustrates the proposed interface projects a 3-D occupancy boxinto the virtual reality stereo goggles, which represents, to theoperator, the goal location of the quadrotor.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the invention of exemplaryembodiments of the invention, reference is made to the accompanyingdrawings (where like numbers represent like elements), which form a parthereof, and in which is shown by way of illustration specific exemplaryembodiments in which the invention may be practiced. These embodimentsare described in sufficient detail to enable those skilled in the art topractice the invention, but other embodiments may be utilized andlogical, mechanical, electrical, and other changes may be made withoutdeparting from the scope of the present invention. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present invention is defined only by the appendedclaims.

In the following description, numerous specific details are set forth toprovide a thorough understanding of the invention. However, it isunderstood that the invention may be practiced without these specificdetails. In other instances, well-known structures and techniques knownto one of ordinary skill in the art have not been shown in detail inorder not to obscure the invention. Referring to the figures, it ispossible to see the various major elements constituting the apparatus ofthe present invention.

The proposed system is a point-and-click control method, where theoperator selects a point in the image for the quadrotor to traverse. Theoperator selects a distance, and the quadrotor will traverse a straight(or curved) line between its current location and the point selected inthe image. In the proposed system, the point selection is performedusing a set of virtual reality stereo goggles (i.e. Oculus Rift). Theoperator is given an omni-directional image, where—by moving his/herhead around—he/she can observe the area surrounding the UAV. The gloveis used as a virtual mouse, selecting the point where the operatordesires the UAV to go, as well as a selecting how far, in thatdirection, the quadrotor should automatically move.

In this modality, the quadrotor will automate a variety of functions forthe operator. The system keeps track of the various poses of the virtualreality stereo goggles; it also keeps track of the location of thepointer, shown three-dimensionally in the image inside those virtualreality stereo goggles. In addition, it keeps track of the pose of thequadrotor from when that image was collected. Finally, by time-taggingand transforming all of those poses, it is capable of figuring out thedirection the quadrotor needs to follow. If the IMU on the quadrotor hasnot drifted, the transformation between the points indicated by theoperator, and the direction of travel of the quadrotor, should beexact—regardless of time drift in the system. This technique borrowsfrom a program that the inventors are currently developing forteleoperating (air, ground, sea, undersea, or in space); in situationswith long communications delay.

Once the operator has indicated the point in space where he would likethe quadrotor to go, he is free to look around through theomnidirectional camera, which would provide significant improvement insituational awareness. Once the vehicle achieves the assigned point, itwill hover until a new point is assigned. In the meantime, while thequadrotor is traversing to the assigned location, the operator canobserve the scene by moving his head.

The operator's selection of the point for the UAV's traversal, isindependent of the intermediate and final poses of the UAV. All thecontrol necessary to turn, rotate, and stabilize the quadrotor in itsway-assigned point is fully automated, and does not need to besupervised by the operator. Aerodynamic effects, created by theproximity to walls or to the ground, are locally counteracted by theon-board control loop.

The advantages of the proposed approach include:

This low-level control loop is significantly faster than thecommonly-used teleoperation loop, which requires video or a sequence ofimages to be sent to the OCU and commands to be returned to the vehicle.Therefore, the system is capable of more rapidly correcting forstability.

The operator does not need to be trained to handle these aerodynmiceffects, which are the common causes of UAV crashes.

By freeing the operator from the usual chores of teleoperation, theoperator has time to acquire better situational awareness of the spacearound the quadrotor.

Even though we are sending the operator an omni-directional image of thespace, these images do not need to be sent at high rates. On thecontrary, we expect these images to be high-resolution, but very lowframe rate—less than 1 hz. Therefore, we also expect this technique touse significantly less bandwidth than teleoperation.

Because time and location tagged images are available at the OCU, theybecome an automated method for collecting the information necessary toenable flashback.

Because the technique does not rely on fast, round-trip communicationsbetween the platform and the OCU to maintain its stability, thetechnique is significantly better suited for situations involvingsignificant delay (multi-hop radius or SATCOM).

Because the proposed technique does not require any expensive or heavysensors, it will still maintain the cost of the quadrotor low.

The challenges of the proposed approach are as follows:

The quadrotor requires an omni-directional camera; or, at least, acamera that provides significantly large field-of-view. For thisapplication, we will leverage smartphone technology that usesinexpensive lenses, and should create fish-eye field-of-views. Theselenses are inexpensive, and available in a variety of sizes and shapes.

Although there are no issues maintaining the Euler angles given theaccuracy of the gyros, there are still issues in double-integrating theaccelerometers, to ensure that the trajectory assigned is correctlyfollowed. A model of motion of the vehicle is used to predict thetrajectory being expressed on the display The unaided, relative positionbetween the starting location and the ending location assigned by theoperator will drift significantly if the double-integration is performedover a significant amount of time. This is not a new problem; InstantEye and other quadrotors zero-out the accelerometer grips by utilizing avariety of tricks. One such trick uses the downward-looking camera andultrasound sensors to zero-out this error. A second approach, utilizedby a variety of short-range missiles, uses the target image in order toguide both the heading and the position of the projectile.

The final challenge is the form factor of the virtual reality stereogoggles or display worn by the operator for the operator. Currentoperators are not used to carrying the stereo goggles, and we will haveto find stereo goggles well-suited to operator environments.Fortunately, the Army is looking at a variety of stereo goggles that canprovide this functionality.

The present invention is based on a virtual reality interface. Thesystem shown in FIGS. 1a-1c is designed to provide situational awarenessto users traveling inside a windowless vehicles. A single,omni-directional camera, installed on top of the ground vehicle,provides the imagery necessary to feed the virtual reality stereogoggles worn by all occupants of the vehicle. As the vehicle traversesthe route towards the mission drop-off point, the users are capable oflooking at the terrain and understanding the scene. Ultimately, when thedoors open and they have to leave the protection of the vehicle, theyminimize the time necessary for understanding their surroundings.

The interface is composed of a few different widgets. As the operatormoves their head, the virtual reality stereo goggles track the differentpositions of all the users' heads in the vehicle. By comparing theseposes with the poses of the omni-directional camera mounted to thevehicle, a real-time stabilization mechanism is utilized, maintainingstable imagery. To the operator, it is as if they are looking out of awindow, without being influenced by the vibrations and motions of thevehicle. The imagery is complimented with maps showing the location ofthe vehicle, as well as the trajectory being followed. As the users lookdown, a top-down view of the map is shown in real time. The pictures100, 101, and 102 from FIGS. 1a-1c show a user looking forward in thedirection of travel 100, then looking lower 101, then looking down 102.One can see the transition between the forward-looking 100 and top viewmaps 103, as depicted in his stereo goggles.

FIG. 2 shows a preliminary concept interface where the operator,utilizing a virtual reality glove, selects a point in the imagery 200presented in the virtual reality stereo goggles. When the operatorselects a point, a box 201, similar to the one shown in FIG. 2, willshow the operator the final location placement of the UAV in 3D space.The interface will also provide the operator with coarse distancemeasurements 202, provided by the acoustic sensors.

To summarize, the operator will find, by moving his head, the locationwhere he would like the quadrotor to go, then use the glove to select apoint in the 3D imagery. That point will indicate the perceived range ofthe location, and draw a 3D prediction of where the quadrotor willtraverse in order to achieve that location. The operator can adjust the“depth” of the traversal; in this case, how close it will get to thewall. Finally, it will press execute. The quadrotor—after the correctintermediate and final pose changes are taken under consideration—willcontrol itself, staying within the corridor indicated in the operators3D stereo goggles. While the traversal is executed, the location of thedesired goal and the perceived distances until collision will be updatedon the operator's visor.

The proposed technique outlines a new control mechanism well-suited forsmall, unmanned aerial vehicles traversing in GPS-denied areas. It hasthe strong advantage of simplifying the interface, so that even anuntrained operator can handle the difficult, dynamic problemsencountered in closed quarters. The proposed system seamlesslyintegrates point-and-click control with way-point navigation, in anintuitive interface. An additional advantage of the proposed system isthat it adds minimal hardware to the payload of the UAV, and canpossibly, strongly diminish the bandwidth and delay effects of thecommunication channel.

Thus, it is appreciated that the optimum dimensional relationships forthe parts of the invention, to include variation in size, materials,shape, form, function, and manner of operation, assembly and use, aredeemed readily apparent and obvious to one of ordinary skill in the art,and all equivalent relationships to those illustrated in the drawingsand described in the above description are intended to be encompassed bythe present invention.

Furthermore, other areas of art may benefit from this method andadjustments to the design are anticipated. Thus, the scope of theinvention should be determined by the appended claims and their legalequivalents, rather than by the examples given.

The embodiments of the invention in which an exclusive property orprivilege is claimed are defined as follows:
 1. A control system devicecomprising: a teleoperated vehicle; an omnidirectional camera or widefield of view mounted on the vehicle; a 3D display worn by the operatorto visualize video or a sequence of images collected by the vehicle; acommunication mechanism between the vehicle and the operator; and adisplay overlay that indicates to the operator the desired goal locationof the vehicle.
 2. The device of claim 1, wherein the 3D stereo gogglesare further comprised of an embedded IMU; and wherein the pose of thehead of the operator can be used to provide “virtual reality”.
 3. Thedevice of claim 1, further comprising a range sensor that covers thedirection of travel as to stop the vehicle from colliding to thelocation assigned by the operator.
 4. The device of claim 2, wherein therange sensor is either acoustic or LADAR.
 5. The device of claim 1,further comprising of a mechanism for rotating the images displayed tothe operator to account for the operator's self motion.
 6. The device ofclaim 1, further comprising a mechanism for rotation the imagesdisplayed to the operator to account for the vehicles motion.
 7. Thedevice of claim 1, further comprising adding the distance to collision,and end estimated time of arrival in the direction indicated by theoperator.
 8. The device of claim 1, wherein the overlays are displayedthree dimensionally by the stereo goggles to express the trajectory andassigned stopping location.
 9. The device of claim 8, wherein a model ofmotion of the vehicle is used to predict the trajectory being expressedon the display.
 10. The device of claim 1, further comprising an inputdevice allowing the operator to select the distance along the trajectorydisplayed in the stereo goggles.
 11. The device of claim 1, furthercomprising the displaying a rendering of the representation of thevehicle.
 12. The device of claim 1, wherein the cameras on the vehicleprovide stereo omnidirectional imagery or generate stereo image pairs byrotating a camera
 13. The device of claim 1, wherein the obstaclessensed by the vehicle are also overlaid on the stereo goggles or otherdisplay.
 14. The device of claim 1, wherein the Euler angles of thedisplay stereo goggles are used to steer the goal location or thetrajectory of the vehicle.
 15. The device of claim 1, wherein multipleoperators can view the imagery and select different goal points.
 16. Thedevice of claim 1, wherein a 2D or 3D top view map showing the locationand trajectory of the vehicle is displayed either as an overlay orutilizing the areas in the hemisphere where the cameras do not cover.17. The device of claim 1, wherein the operator can rewind the imageryto a previous time and still select the goal point.
 18. The device ofclaim 1, wherein a top view display of the obstacles found by thevehicle are displayed on the 3D stereo goggles.
 19. The device of claim1, further comprising a navigation unit that controls or guides thevehicle to the assigned location.
 20. The device of claim 1, furthercomprising a depiction of the final pose of the system and a method forselecting this final pose or intermediate poses.
 21. A control systemdevice comprising: an unmanned air or ground vehicle; an omnidirectionalcamera or wide field of view mounted on the unmanned vehicle; a set of3D stereo goggles that displays a series of images collected on theunmanned vehicle; a communication mechanism between the vehicle and theoperator; and a display overlay that indicates to the operator thedesired goal location of the vehicle.